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Abstract 


This survey paper uses a framework derived from Herman and Winters 
( 1 994) to analyze seven selected research studies for evidence of electronic 
por folios’ technical quality, implementation effects, fairness, feasibility, 
and tool effects. Analysis sought to determine what methodologies and 
instruments for data collection yielded useful information about electronic 
porfolios, and what the findings and methods suggest about directions 
for future e-por folio research. The author’s recommendations are placed 
in the context of an educational technology research agenda proposed by 
leaders in the field. 


P ortfolios have become a widely-used method of assessment in 
American education — particularly for teachers. Nearly 90% 
of schools, colleges, and departments of education (SCDEs) 
use portfolios to make decisions about candidates (Salzman, Denner, 
& Harris, 2002). School districts are using portfolios in mentoring 
programs for novice P-12 teachers (Macisaac, 1992; Whitaker & Ray, 
1994) and for professional development of more experienced teachers 
(Wolf Lichtenstein, Bartlett, & Hartman, 1996). States and national 
professional organizations have begun experimenting with portfolios 
as ways to assess and certify exemplary teachers (Dollase, 1996; Izu, 
Long, Stansbury, & Tierney, 1995; National Board, 1989). Portfolio 
assessment of student learning is also common in P-12 schools (Dol- 
lase, 1996). 

A Dearth of Empirical Research 

Portfolio proponents cite strong theoretical support for their use by both 
teachers and students, arguing they are good devices for capturing evidence 
of knowledge in context and prompting new learning (e.g., Shulman, 
1987, 1992, 1998; Wiggins, 1989; Yancy, 1992). Unfortunately, em- 
pirical evidence documenting the effects of portfolios is sparse. In 1994, 
Herman and Winters noted the supposed advantages of portfolios for 
learning and assessment, but pointed out deficiencies in the research: “Of 
89 entries on portfolio assessment topics found in the research literature 
over the past 10 years, only seven articles either report technical data 
or employ accepted research methods. Relatively absent is attention to 
technical quality, to serious indicators of impact, or to rigorous testing 
of assumptions” (p. 1). 

Has the research situation improved since 1994? Not greatly, it ap- 
pears. Lyons noted in 1998: “There is not yet a body of systematic data 
documenting their [portfolio] uses or their long-term consequences” (p. 
247) . More recently, Zeichner and Wray (2001) voiced the same concern: 
“Despite the current popularity of teaching portfolios, there have been 
very few systematic studies of the nature and consequences of their use 
for either assessment or development purposes” (p. 615). 


Are portfolios effective devices for assessing and developing knowl- 
edge, as proponents claim? We have little empirical evidence to make 
the case. 

Expanding Use of Electronic Portfolios 

As the portfolio becomes an ever more prominent device for assessment 
and learning, advances in technology are radically changing its form. Once 
predominantly paper text, portfolios are now more likely to be digital 
products presented using the computer — either by means of off-the-shelf 
tools or server-based software. In higher education, teachers and students 
are increasingly being given access to electronic portfolio systems on 
institution servers. According to the 2004 Campus Computing Project 
survey (Green, 2004), electronic portfolios are currently provided by 
nearly 30% of public universities and 18% of private universities across 
the country. 

The use of electronic portfolios has become even more common in 
teacher education. Colleges of education, under pressure by state and 
other accrediting agencies to document students’ standards-based com- 
petencies, have embraced the technology. One sign of their interest is 
that 50% of the 400 Preparing Tomorrow’s Teachers to Use Technology 
(PT^) programs have used grant funds in part to develop and implement 
electronic portfolio systems (Britten, Mullen, & Stuve, 2003). 

How do these emerging digital technologies affect portfolio product 
and process? Although we have a host of preliminary reports about 
implementation in teacher education programs, there have been very few 
systematic studies to probe assumptions about the device and provide 
empirical evidence of electronic portfolios’ effectiveness for assessment 
or learning. 

Concerns and Questions about Electronic Portfolios 

Despite their increasing popularity, a host of questions and concerns have 
been raised about electronic portfolios. Maddux and Cummings (2004) 
have suggested that electronic portfolios might be one of the current hot 
topics susceptible to being caught up in what they refer to as the pendulum 
syndrome. This term refers to the tendency for many a new educational 
practice or technology to be greeted with overly optimistic claims; then 
when the innovation fails to live up to unrealistic expectations, it is 
abandoned — often before it has even been fully or appropriately imple- 
mented. Efforts to validate the innovation through research are often still 
at preliminary stages when its wave of popularity ends and the practice 
is discarded. We might ask ourselves: In the absence of research to guide 
their use, will electronic portfolios be an educational fad destined to go 
the way of Papert’s Logo turtles? 

Developments in software also prompt new questions and concerns. 
Although most electronic portfolios were initially constructed with 


Volume 22 / Number 3 Spring 2006 Journal of Computing in Teacher Education 89 



off-the-shelf productivity or Weh authoring software, their nature is 
now being transformed hy specialized e-portfoIio software. In teacher 
education especially, many electronic portfolios are now constructed 
with commercial database software residing on an institution’s server 
(e.g., LiveText, Chalk and Wire, Taskstream). These portfolios are often 
highly structured and used for large-scale assessment and program evalu- 
ation. Barrett (2004) argues that in many cases, these products should 
more properly be called assessment management systems rather than 
electronic portfolios. What are the implications of using portfolios for 
such high-stakes assessment? 

If portfolios are going to be used to make important screening and 
placement decisions about teachers or students, we need to be very careful 
that they meet the technical standards for any good measurement device. 
Do we have empirical evidence that portfolios can be scored reliably and 
that they enable us to make valid interpretations of a candidate’s com- 
petencies? And even if portfolios can be made to function in this way, 
is it wise to use them in such a manner? If we use portfolios to make 
high-stakes decisions, will we have destroyed their usefulness as a learning 
tool — making them what Shulman (1998) has referred to as a “very, very 
cumbersome multiple-choice test” (p. 35). 

These are but a few of the questions that have been raised about electronic 
portfolios — and few answers are to be gleaned from published research. 

Limited Research on Electronic Portfolios 

In examining portfolio literature one soon notes that most articles and pa- 
pers on the subject are conceptual or anecdotal rather than research-based. 
Many are implementation reports describing the features of particular 
portfolio programs, frequently accompanied by survey or interview data 
on the attitudes and beliefs of portfolios authors. Although description 
of the manner in which portfolios are implemented can be useful in the 
design of other portfolio systems and can identify promising areas for 
research, this type of inquiry in and of itself will not make the case for 
portfolios as effective assessment or learning devices. 

Many studies offer self-report data from authors who testify to the 
learning benefits of the portfolio; researchers must be careful not to stop 
there. Additional data are needed to establish the veracity of portfolio 
authors’ perceptions. If the purpose of a portfolio is to foster learning, hard 
evidence that the portfolio has indeed advanced the author’s knowledge 
must be sought. If the primary purpose of a portfolio is assessment, we 
need confirmation that an evaluator is able to use portfolio evidence 
to accurately assess a candidate’s skills and knowledge — whether that 
argument is based upon principles of psychometrics or hermeneutics 
(Moss, 1998). 

National Electronic Portfolio Research Initiatives 

The many questions about electronic portfolios and scarcity of research 
have prompted new national efforts to study them. One noteworthy 
example is the National Coalition on Electronic Portfolio Learning, coor- 
dinated by the American Association of Higher Education and Clemson 
University’s Pearce Center. This three-year project enlists the efforts of 
faculty from 30 higher education institutions to design research on digital 
portfolios’ effects on learning. 

The use ofWeb-based electronic portfolios among high school students 
is the focus of the Researching Electronic Portfolios: Learning, Engage- 
ment, Collaboration through Technology (REFLECT) initiative, led by 
Helen Barrett and underwritten byTaskStream. This 18-month national 
action-research project will study the effect of electronic portfolios on 
high school students’ learning, motivation, and engagement. 

Large-scale research is also underway to understand the diffusion of 
e-portfoIios in schools, colleges, and departments of education (Strudler 
& Wetzel, 2005). Beginning with a survey completed by 23 institutions 


thought to be mature users of the technology, Strudler and Wetzel chose 
six institutions from across the country for in-depth study. Data collection 
includes site visits and more than 80 interviews. 

In addition to these national efforts, many individuals and groups 
plan their own research on electronic portfolios. What direction should 
this research take, and what methods would be most productive in 
studying electronic portfolios? I felt that past research might provide us 
with some clues. 

What Does Past Research Tell Us? 

Looking for e-portfolio studies that might guide the design of new 
research, I did a new review of portfolio literature, paying particular 
attention to studies of preservice and inservice teacher portfolios. My 
objective was to find a small group of studies that used rigorous methods 
and displayed the features called for by Herman and Winters (1994): “at- 
tention to technical quality, to serious indicators of impact, or to rigorous 
testing of assumptions” (p. 1). 

From a body of approximately 25 empirical studies of electronic port- 
folios, I selected six for analysis of methodology and findings. One study 
of traditional-format portfolios was included in the analysis because it 
complements one of the e-portfolio studies and sheds light on how student 
portfolios might be used for teacher professional development. In choosing 
these particular studies for analysis, I do not mean to suggest they are superior 
to others done during the period — there is a certain arbitrary element to my 
choices, and, undoubtedly, there are numerous high-quality studies of which 
I am unaware. These were chosen because I felt each revealed something 
useful about portfolios and about methodologies for studying them. 

Using a framework derived from Herman and Winters (1994), 
but adapted for portfolios in a digital format, I looked for evidence of 
electronic portfolios’ technical quality, implementation effects, fairness, 
feasibility, and tool effects. 

Analysis was guided by these questions: How did particular studies 
use theoretical frameworks to ask important questions about portfolios? 
What methodologies and instruments for data collection yielded useful 
information about electronic portfolios? What might these studies tell 
us about directions for future e-portfolio research? 

In this article I discuss the methodology of each study, highlighting 
particular aspects of their design and summarizing major findings. In 
the Conclusion section I use analysis of the selected studies to suggest 
directions and methods for electronic portfolio research, placing my 
own ideas in the context of an educational technology research agenda 
proposed by leaders in the field. 

Analyzing Selected Research on Teacher 
Portfolios 

Herman and Winters ( 1 994) argued that we need evidence in four main 
areas to certify the soundness of portfolio assessment: (a) technical quality, 
(b) implementation effects, (c) fairness, and (d) feasibility. I have adapted 
these characteristics to apply not only to assessment portfolios, but also 
to those whose primary purpose is learning. 

I added one additional category for analysis of research on digital 
portfolios. In investigating electronic portfolios, the manner in which 
the technological tool affected electronic portfolio process and product 
ought to be considered. I call this feature tool effects. In evaluating tool 
effects, one ought to begin with the question: What unique contribution 
did this tool make? Did the technology enable us to do something that 
we could not achieve without it? 

These five criteria provide a framework for closely examining selected 
portfolio research. I begin my analysis with four studies that shed light on 
the technical quality of electronic portfolios; discussion of research relat- 
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ing to portfolio implementation effects, fairness, and feasibility follows. 
Consideration of tool effects is interwoven in each category. 

Evidence of Electronic PortfolioTechnical 
Quality 

What do we mean hy technical quality in portfolios? Herman and Win- 
ters (1994) identify the two core elements of psychometrics — reliability 
and validity — as key indicators of quality when referring to assessment 
portfolios. If on the other hand, a portfolio’s primary purpose is to foster 
learning, I suggest technical quality would lie in its capacity for prompting 
and structuring deep learning. 

What do we know about the technical quality of electronic portfolios? 

I begin with a study by Derham (2003) that investigated factors of most 
concern for assessment portfolios — their reliability and validity — then 
focus on three studies of electronic portfolios used primarily for learning 
purposes: Avraamidou and Zembal-Saul (2004), Hartmann (2003), and 
a related study by Hartmann and Calandra (2004). 

Technical Quality in Assessment Portfolios 

A limited number of studies have considered the reliability and validity 
of traditional-format student portfolios (e.g., Gearhart, Herman, Baker, 
& Whittaker, 1993; Herman & Gearhart 1993; LeMahieu, Gitomer, & 
Eresh, 1995; Supovitz & Brennan 1997; Supovitz & MacGowan, 1997); 
the reliability and validity of electronic portfolios has yet to be established. 
A study by Derham (2003) is an important step in that direction. 

Methodology. Garol Derham’s dissertation research was an investiga- 
tion of the reliability and validity of a newly-developed digital teaching 
portfolio, the Digital Portfolio Assessment of Teaching Gompetencies 
(D-PATGO). This portfolio was designed to assess the instructional 
competencies of preservice teachers. Derham examined evidence of the 
D-PATGO’s psychometric properties based on a number of factors: test 
content (measured by expert opinion), relations with other variables (other 
preservice teacher assessments), and reliability estimates (internal consis- 
tency and interrater reliability). Data were collected from 30 preservice 
teachers during the course of four semesters and analyzed using multiple 
methods, including Pearson product-moment correlation coefficients, 
Gronbach’s alpha, and Gohen’s kappa coefficient. 

Findings. Findings indicate that assessment of instructional com- 
petence is possible using a digital preservice teacher portfolio. Derham 
found that although D-PATGO demonstrated theoretically acceptable 
relationships with several other assessments of teacher competency and a 
generally positive expert review, there were weaknesses in the D-PATGO 
assessment tool: low inter-rater reliability and inadequate evidence of 
preservice teachers’ content knowledge. 

Derham reports that four key conclusions can be drawn about port- 
folios: First, it is feasible to develop and implement a digital portfolio 
that is adequate in assessing a majority of widely adopted standards for 
beginning teachers. Second, the D-PATGO assesses pedagogical knowl- 
edge and skills in a way comparable to grades in methods courses and 
scores on Praxis II. Third, this measure does not address the full breadth 
of preservice teachers’ content knowledge. 

Research such as Derham’s helps us determine the technical quality of 
assessment portfolios. Avraamidou and Zembal-Saul (2004) shed light 
on how electronic portfolios can be used for learning. 

Technical Quality in Learning Porfolios 

Avraamidou and Zembal-Saul studied the Web-based electronic portfolios 
of two prospective elementary science teachers. Findings in this study 
help us verify that an electronic portfolio can foster deep learning and 
indicate conditions of significance in structuring it. Their methodology 
may be useful for other researchers to consider. 
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Methodology. Research was guided by theories suggesting that port- 
folios could be used to support preservice teachers’ reflective thinking. 
Avraamidou and Zembal-Saul theorized that the Web format would 
facilitate complex interconnections between participants’ personal phi- 
losophies for teaching science and multimedia evidence used to support 
those claims. 

Multiple sources of data were used to investigate the nature of 
preservice teachers’ understandings of the teaching of science and the 
manner in which task and technology supported thoughtful reflection. 
Two types of data were drawn from participants’ Web-based portfolios: 
three versions of participants’ philosophies for teaching science developed 
over time, and the reflective statements that accompanied each (i.e., 
participants discussed what changes were made in the different versions 
and explained why). 

Three techniques were used to analyze the data: pattern-matching, 
explanation-building, and time-series analysis (Yin, 1984) as well as 
content analysis done on the reflective statements. The way participants 
used the capabilities of the Web format for displaying multimedia and 
hyperlinking was also carefully investigated. 

Findings. Data analysis revealed evidence of learning and professional 
development in three areas: 

• Making connections between university coursework and field expe- 
riences. 

• Developing reflections from descriptive to explanatory 

• Engaging in reflective and metacognitive activities 

The researchers also reported that participants’ portfolios displayed 
growth in pedagogical content knowledge for teaching science: participants’ 
philosophies demonstrated that they had become more sensitive to children’s 
thinking, to connecting physical engagement of children with conceptual 
aspects of learning, and focusing on teaching science as inquiry. 

Avraamidou & Zembal-Saul make an interesting connection between 
the nature of the task and the affordances of the technology: (a) hyper- 
linking is used by portfolio authors to make non-linear, dynamic repre- 
sentations of their science teaching philosophy, and (b) the public nature 
of Web publishing makes thinking visible to a large audience — which 
motivates portfolio authors to produce their best work and enables them 
to give and receive feedback from a wide audience. These tool effects could 
not have been achieved without Web technology. 

What is going on cognitively as teachers reflect on theory and practice, 
as they did in Avraamidou & Zembal-Saul’s study? Ghristopher Hartmann 
(2003) provides us with a conceptual framework for teacher cognition 
and more evidence of what features of task and setting are important for 
developing professional “habits of mind.” Hartmann’s study is yet another 
example of the kind of systematic research we can use in documenting 
the technical quality of learning portfolios. 

Methodology. In dissertation research, Ghristopher Hartmann (2003) 
investigated how seven prospective teachers of secondary mathematics 
learned to represent their teaching practice in an electronic portfolio. Hart- 
mann uses Goodman’s (1978) theory of rendering as a conceptual frame- 
work for understanding the process of teacher reflection on practice. 

Data were gathered over two semesters of portfolio work from mul- 
tiple sources: two semi-structured interviews, one focus group interview, 
observation of a portfolio seminar presentation, and three versions of the 
portfolio collected at different times during the process. Using “critical 
incidents of practice” as the unit of analysis allowed Hartman to trian- 
gulate across different data sources — each incident having been chosen 
for analysis because it appeared during an interview and also in the form 
of portfolio evidence. Analysis was by means of constant comparative 
methodology. Glose examination of portfolio content was central to his 
investigation. 
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Findings. Hartmann’s findings suggest that learning to render one’s 
practice is cognitive scaffolding for preservice teachers as they develop 
the hahits of mind necessary for them to grow toward high-quality 
mathematics instruction. Powerful learning occurred when participants 
were asked to render a single lesson multiple times (a finding similar to 
Avraamidou and Zemhal-Saul). 

Hartmann also suggests that establishing the portfolio as the beginning 
of a professional continuum of rendering and sharing one’s practice gave 
portfolio authors an intrinsic purpose for portfolio authoring — a factor 
important for preservice teachers to make connections between their 
teaching experiences and university coursework. He emphasizes that 
much of the power of the portfolio in his study was due to the manner 
in which it was situated in a professional learning community. 

In collaboration with Calandra, Hartmann did additional analysis of 
his corpus to investigate how technology impacted learning in a commu- 
nity of practice (Hartmann & Calandra, 2004). This analysis enabled the 
researchers to document how portfolio design fostered the development of 
community among the seven participants and how technological innova- 
tion disseminated through the group. Hartmann and Calandra point out 
how technological innovations enhanced portfolio authors’ capability for 
representing their teaching of mathematics, and document tool effects. 

Summary: Evidence of Technical Quality 

The three studies analyzed provide evidence of electronic portfolios’ 
technical quality as assessment and learning devices. Can portfolios be 
used to assess teachers’ knowledge and skills with reliability and validity? 
Derham’s findings suggest they have that capability. Can portfolios prompt 
deep learning? Avraamidou and Zembal-Saul ‘s research provides evidence 
that they do: “Our findings strongly suggest that prospective elementary 
teachers’ learning could be enhanced through the Web-based portfolio 
development, which engages them in reflective and meta-cognitive activi- 
ties about their views of science teaching and learning” (p. 438). 

How can electronic portfolios prompt learning? Hartmann’s findings 
indicate they do so most powerfully when constructed in a learning 
community. Both studies suggest that engaging preservice teachers in a 
recursive task requiring them to justify their beliefs, values, or practices 
with evidence from theory and field experience prompts reflection and 
results in significant learning. Features of electronic portfolio technology 
can support learning community and structure meta-cognitive activity. 

What do these studies tell us about methods for researching electronic 
portfolios? Although methodology varied, each study defined clear re- 
search questions grounded in theory. Data were collected systematically 
from multiple sources and triangulated to establish its trustworthiness 
(Lincoln & Cuba, 1985). Portfolio content was carefully analyzed. 

Evidence of Electronic Portfolio 
Implementation Effects 

Herman and Winters called for research to document the effects of portfolio 
implementation. Two studies of portfolio authoring in P-12 settings present 
some interesting findings about the effects of portfolios when elementary 
students and their teachers are engaged in the process of portfolio author- 
ing. These studies are of note methodologically because they were located 
in P-12 classroom contexts, were conducted over an extended period (1-3 
years) and use observational data in addition to interviews. 

Implementation Effects of Learning Portfolios 

Mumbi Kariuki (2001) studied preservice teachers who had been provided 
with laptop computers for use during a year-long internship experience 
in an elementary school. Each preservice teacher worked together with an 
elementary pupil to develop an electronic portfolio for the pupil. 


Methodology. Fifteen preservice teachers, two mentor teachers, and 
fifteen elementary students participated in the Kariuki study, which lasted 
for the duration of a school quarter. Participant observations and in-depth 
interviews were conducted to investigate how the preservice teachers used 
the laptops and the effect of this use. 

Findings. Findings in this study were that laptops gave preservice 
teachers quick access to technology, provided them with an opportunity 
to develop confidence in the integration of technology into teaching, 
and enabled them to become co-learners with their students. Elementary 
pupils also benefited: they displayed greater enthusiasm and motivation for 
learning; they had an opportunity not only to develop technology skills, 
but also to take on a mentoring role with an adult. Construction of an 
electronic portfolio by students and preservice teachers in an elementary 
classroom also provided the students’ own teachers with a model for how 
technology can be used, enhancing their professional development. 

The findings suggest that providing an opportunity to use technology 
with elementary pupils in a non-threatening setting is one way to prepare 
teachers who can integrate technology in their own classrooms. The study 
recommends the use of a project-based approach, such as electronic port- 
folios, when preservice teachers are provided with access to technology. 
In this case, the tool made a difference — it helped both preservice and 
inservice teachers develop new technology competencies and enabled 
both teachers and students to construct e-portfolios. 

Although Judith Ellsworth (2002) did not do research on electronic 
portfolios, I include them in this analysis because her methods and find- 
ings are similar to Kariuki (2001): they shed additional light on the effects 
of implementation in a P-12 setting and show how student portfolios 
can result in teacher learning. Ellsworth’s study is sensitive to the systemic 
effects of portfolio adoption in a school. 

Methodology. Ellsworth’s research was a three-year case study of an 
elementary school in which student portfolios were implemented as part 
of a comprehensive school reform effort. The study documents the specific 
role of portfolios in helping teachers at the site confront the discrepan- 
cies between what they thought they had taught, and what students had 
actually learned. The study also addressed the larger issues of teachers’ 
changing conceptions of instructional practices and professional develop- 
ment needs resulting from a reform process. 

Research was based on a phenomenological perspective, which seeks 
understanding from the perspective of the participants and is based pri- 
marily on descriptive data. Data sources included participant observation 
and in-depth interviews. 

Findings. Findings indicate that portfolios can help teachers come to a 
deeper understanding of their professional practices. In this case, teachers 
gained a more comprehensive view of student learning by means of their 
portfolios. Teachers reported that no other form of assessment provided 
them with as much information for reflection. 

Researchers noted that evidence from student portfolios was used by 
teachers to analyze and improve their instructional practices. Teachers 
were observed questioning their previous assumptions about teaching and 
learning; they became more deliberate in questioning the rationale for 
their instructional choices. Teachers also gave up some classroom control 
and began to conduct portfolio conferences with students and parents. 

Ellsworth considers the greater involvement with parents to be a sig- 
nificant benefit of the portfolio: “Teachers recognized that the parents’ 
reflection, along with the student’s and the teacher’s, created a more 
complete portrait of the student learner and provided another opportunity 
for teachers to reflect on instructional practices” (p. 351). 

Portfolios had an effect on the school-wide reform effort: teachers began 
to make changes in classroom practice, to take on new school- wide respon- 
sibilities, and to take charge of their own professional development. 
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Summary— Portfolio Implementation Effects 

Kariuki (2001) and Ellsworth (2002) contribute to our understanding of 
how P-12 student portfolio implementation can result in teacher learning. 
The location of this research in classroom settings over extended periods 
of time, the use of ethnographic methods, and collection of observational 
data make these studies worthy of note, and suggest productive areas 
for further research. We need more empirical research of this type to 
identify the effects of portfolio implementation in various settings, and 
with various tools. 

Evidence of Electronic Portfolio Fairness 

A third category Herman and Winters identified as important for research- 
ing portfolios is that of fairness. By this they mean. Whose work is it? Her- 
man and Winters suggest that the amount of support a portfolio author 
receives might cast doubt on the fairness of the portfolio as an assessment 
device. If we are trying to establish what a given individual can do, widely 
differing amounts of assistance from peers, instructors, and others could be 
problematic. This issue is particularly significant when portfolios are used 
for high-stakes decision making about individuals or institutions. 

This issue of Whose work is it? can be conceptualized in a different 
manner, however. Wineburg (1997) and others (e.g., Shulman, 1998) 
have argued that teacher assessment should occur in a collaborative 
context and that portfolio authoring ought to be a coached activity. The 
National Board for Professional Teaching Standards portfolio was delib- 
erately structured to promote this kind of teamwork. Hartmann (2003) 
provides evidence that electronic portfolios can be made more effective 
by situating them in a professional learning community. 

The concept of portfolio fairness also means something additional: Does 
the portfolio result in patterns of high or low performance for individuals 
of different genders or different socio-economic, ethnic, or racial groups? 
Wilkerson and Lang (2003) have explored the ramifications of this issue 
when portfolios are used as high-stakes assessments. They point out that 
portfolios must be shown to have no bias toward identifiable subgroups 
of the population, must allow for equitable treatment in the “testing” 
process, and must provide opportunities for all to learn whatever skills 
and knowledge are required for a satisfactory portfolio performance. If 
electronic portfolios are to be used for large-scale assessment purposes, we 
need research to verify that scores are not biased against students of different 
genders, SES status, race, culture, or even academic background. 

Fairness in Assessment Portfolios 

Derham’s study (2003) indicates two areas of concern for e-portfoIio fair- 
ness. She notes that programs wishing to use the D-PATCO and other 
digital portfolios will need to address the technological preparedness of 
their students if they wish to ensure fair and equitable assessment. Students 
will need not only the technological skills to author the portfolio, but 
also ready access to ancillary hardware and software. Derham found that 
participants in her study did not make use of lab resources for construct- 
ing their portfolios, but instead preferred to work at home. This is an 
issue not only of skill, but of economic resources as well. Will electronic 
portfolios disadvantage lower SES students? 

We need additional research to answer this question and identify other 
factors that might interfere with equitable assessment. Because e-portfolio 
software differs so greatly in the level of technology skill required, research 
to determine what supports are needed for students to equitably construct 
e-portfoIios must be sensitive to tool effects. 

Fairness in Learning Portfolios 

Empirical research is also necessary to identify what types of cognitive 
scaffolding and programmatic features are important for all students 
to benefit equally when a portfolio program has learning as its primary 
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purpose. The many studies describing implementation of electronic 
portfolio programs in teacher education, if synthesized, could provide 
us with knowledge about how particular context variables interact with 
various e-portfolio tools. 

Summary— Electronic Portfolio Fairness 

Providing evidence of the fairness of electronic portfolios for assessment 
and learning will first require a conceptual understanding of the role of 
collaboration in portfolio authoring. Meta- analysis of portfolio assessment 
data could determine if there are patterns of high or low performance for 
individuals of different genders or different socio-economic, ethnic, or 
racial groups. Qualitative research is needed to document the features of 
tool and program that result in particular types of learning. 

The extensive body of data accumulated by PT^ projects with electronic 
portfolio programs could provide valuable information about fairness in 
assessment and learning portfolios. A summit of PT^ leaders (Thompson, 
2005) recently called for a synthesis of knowledge from PT^ evaluation 
data and for the design of long-term, cross-project instrumentation for 
metadata analysis. Additional research on NBPTS portfolios might also 
help us understand the role of portfolio support materials and professional 
collaboration for electronic portfolios. 

Evidence on Electronic Portfolio Feasibility 

Herman and Winters (1994) called for research in one final category: 
portfolio feasibility. They identified a host of challenges making portfolios 
difficult to implement successfully in P-12 contexts: 

• Portfolios make substantial demands on teacher and student time 

• Teachers must learn new assessment practices 

• Appropriate portfolio tasks and rubrics must be devised 

• Portfolios require a culture of inquiry and student reflection on 
their own learning processes 

• Teachers must take on new roles 

Many of these challenges apply also to teacher education. 

Electronic portfolios raise additional issues: Do portfolio authors 
have access to sufficient technology? Is technical support available and 
adequate? Do portfolio authors have necessary skills for using the hardware 
and software? And finally, what will it cost to provide all of these supports? 
Electronic portfolios will not be feasible for assessment or learning unless 
these issues are addressed. 

Wilkerson and Lang (2003) have alerted teacher education programs to 
a host of psychometric, legal, and practical issues that must be faced when 
one seeks to use electronic portfolios for assessment and certification of 
teachers. My own research on preservice teacher portfolios (Carney, 2001, 
2002, 2003) suggests other issues and dilemmas. Electronic portfolios 
may not prove feasible as devices for assessment or learning if we do not 
recognize their many challenges. 

Feasibility of Assessment Portfolios 

The feasibility of using portfolios for assessment may depend on how 
successfully we deal with a host of psychometric, legal, and practical fac- 
tors. When portfolios are used for large-scale assessment, as they often are 
in teacher education, feasibility becomes a pressing concern. Wilkerson 
and Lang (2003) point out that portfolios used for certification or other 
high-stakes decision making must be psychometrically sound, or schools, 
colleges, or departments of education may be subject to legal challenges. 
Although not an empirical study, Wilkerson and Lang’s cautions are 
worthy of careful consideration for the design of research on portfolios 
used for high-stakes assessment. 

Wilkerson and Lang (2003) cite arguments from the literature, point- 
ing out the problems associated with portfolios in a high-stakes testing 
environment: validity, reliability, fairness, excessive time burdens, and 
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portfolios’ loss of value as a means to improve learning. Although they 
identify eight requirements for the use of portfolios as certification “tests,” 
Wilkerson & Lange suggest that portfolios may be more appropriately 
used for formative assessment in support of learning rather than as a 
summative certification device. They assert that the most feasible use of 
portfolios for teacher assessment may be in amassing artifacts of P-12 
student work to demonstrate a teacher’s capacity for designing effective 
instruction. 

Much more empirical research will be needed to make the case for 
electronic portfolios as summative assessment devices. 

Feasibility of Learning Portfolios 

Is it feasible to use electronic portfolios for preservice and inservice teach- 
ers if their primary purpose is learning, and assessment is formative? My 
own research (Carney, 2001) provides some evidence of how electronic 
portfolios function in that capacity. 

Research was designed as a collection of in-depth studies of the port- 
folios completed by six candidates for certification as secondary teachers 
in a Masters in Teaching program. At the time they were engaged in 
portfolio authoring, participants had just completed student teaching 
and were in the final quarter of their program. 

Although the portfolio at this university was a degree requirement 
assessed by means of a tri-Ievel rubric, its primary purpose was learning, 
and assessment was formative: all students were expected to achieve at least 
a Standard level on the assessment rubric and had multiple opportuni- 
ties to re-do work until it was considered satisfactory. While assembling 
their portfolios, students participated in a portfolio seminar where they 
received support and regular feedback from the seminar instructor, peers, 
and a teaching assistant assigned as mentor. A key portfolio goal was to 
structure reflective thinking. 

Methodology. Research was an effort to understand how preservice 
teachers conceptualized themselves, represented their knowledge, and 
communicated it to others by means of electronic and traditional port- 
folios. The study’s framework was drawn from theories conceptualizing 
teacher knowledge (Shulman, 1986, 1987) and tool-based socio-cultural 
activity (Vygotsky, 1981). I considered how the tool chosen for portfolio 
authoring interacts with other artifacts in the setting to influence authors’ 
sense of audience and purpose, and how audience and purpose affected 
portfolio form and content. Artifacts within the portfolios were carefully 
analyzed for teachers’ pedagogical content knowledge (Grossman, 1990; 
Shulman, 1986, 1987; Wilson, Shulman, & Richert, 1987). 

Case study methodology was used to develop six cases; they included 
both a paper and an electronic portfolio from three different secondary 
subject areas: language arts, social studies, and science (physics). Data were 
collected by means of participant observation, think- aloud commentaries, 
participant interview, and examination of the Teacher Education Program 
portfolio requirements and rubrics. 

Data analysis was ongoing, as recommended by Miles and Huber- 
man (1994). Each individual case was reviewed and coded into units of 
meaning, beginning first with descriptive and interpretive coding, and 
working finally into pattern coding (Miles & Huberman, 1994). Two 
separate cross-case analyses of the data were subsequently completed, 
comparing the two participants within each subject matter in two groups 
based on whether they had used traditional or electronic tools for port- 
folio authoring. The manner in which technological tools were used to 
accomplish particular representational or cognitive activities was carefully 
investigated and described. 

Findings. Primary findings include the following assertions: 

1 . Trying to achieve multiple purposes with a portfolio can be 

problematic if it forces the author to write for multiple audiences 


with different information needs and beliefs. Portfolios written for 
multiple audiences were less successful in achieving their purposes 
than were portfolios written for one audience. The exception was 
when audiences were perceived to have similar beliefs and goals. 

2. Defining a potentially critical audience can inhibit preservice teach- 
ers from honestly representing themselves as teachers and engag- 
ing with the problems they faced as student teachers. Making the 
portfolio a high-stakes assessment or suggesting that it might be a 
device for obtaining a job makes it unlikely that the portfolio will 
be used for deep reflection on practice. 

3. Portfolios can provide a vehicle for documenting teachers’ pedagog- 
ical knowledge. How teachers transform subject matter knowledge 
into curriculum, representations, and instructional strategies for 
teaching can be demonstrated in portfolio artifacts and reflective 
entry slips. Differences in PCK indicative of differing levels of 
teacher knowledge can be documented. Enunciating a purpose for 
teaching one’s subject may be as important a theoretical stance as 
formulating a philosophy of education. 

4. If we wish all aspects of pedagogical content knowledge to be rep- 
resented in portfolios, including strategies for technology integra- 
tion, portfolio recipes must properly scaffold the activity. Portfolio 
authors were focused more on teacher action than student learn- 
ing. With little classroom experience, this category of pedagogical 
content knowledge is likely to be deficient; however, the consistent 
lack of reflection on individual student learning suggests portfolio 
recipes failed to structure it. 

5. The manner in which a tool mediates activity is a complex process 
of individual and social interaction. Any program that seeks to use 
portfolios as devices to document teacher knowledge and promote 
reflective practice must attend to the manner in which technologi- 
cal and psychological tools are used. Both tools and people in the 
setting interact to shape portfolio form and content. Small features 
of each tool, in concert with personal interactions, can have a pow- 
erful effect. 

6. There are significant differences between traditional paper and 
electronic Web portfolios. Presentation devices have distinctive pro- 
files based on the affordances and constraints of the tool (Gibson, 
1979). Many of its affordances of the electronic medium are not yet 
being used to advantage. One of the most notable affordances of 
electronic portfolios seems is their capacity for continuing profes- 
sional development. 

Summary— Electronic Portfolio Feasibility 

The use of electronic portfolios for teacher assessment and learning may 
depend upon how successfully we recognize and deal with the many 
psychometric, legal, and practical issues they entail. Evidence suggests 
portfolios may be more feasible as formative assessment devices in support 
of learning rather than for summative assessment, features of electronic 
portfolio software can interact with other tools in the setting to shape 
portfolio form and content in subtle and unexpected ways — which adds 
to the complexity of the planning process, and the necessity for research 
sensitive to contextual variables. 

Conclusion 

What can we learn from the seven empirical studies examined here? How 
did these studies use theoretical frameworks to ask important questions 
about portfolios? What methodologies and instruments for data collec- 
tion yielded useful information about electronic portfolios? What do they 
suggest about directions for future e-portfoIio research? Insights from my 
analysis are communicated in the form of five assertions below. In mak- 
ing these statements, I also draw upon recommendations by educational 
technology and teacher education leaders. 
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Assertion 1 : Important questions about portfolios should be derived 
from robust theories of cognition, learning, motivation, tool use, and 
other relevant concepts. 

Whether quantitative or qualitative methods are used, research on 
electronic portfolios ought to be grounded in theory. Several of the studies 
analyzed in this article are clearly grounded in theory (e.g., Carney, 2001; 
Hartmann, 2003): research questions arise from theoretical issues, and 
findings are related back to theory. Other studies may have been theory- 
driven, but published reports failed to make the connections clear. 

In calling for a new millennium research agenda on educational 
technology, Robyler and Knezek (2003) remind us: “It is axiomatic 
that educational research should help address heretofore unresolved or 
poorly resolved educational problem(s), and that hypotheses should be 
derived from a theory base (p. 67).” In researching electronic portfolios, 
we must heed Maddux and Cummings’ (2004) warning about fads in 
educational technology. 

Assertion 2: Research on electronic portfolios ought to probe basic 
assumptions and claims about both portfolios and technology. 

Studies analyzed here investigate the technical quality of portfolios, 
probing the claims of theorists who have proposed portfolios as devices 
for assessment and learning. Findings document that electronic portfolios 
can function as theorized, and indicate conditions for how they might 
achieve their purposes. 

Many research studies of electronic portfolios have focused on 
methods of implementation, ignoring the question of why we should 
use the technology in the first place. In surveying the field, Nordrum, 
Shitaoka, and Steel (2005) note: “E-portfolios as they are currently used, 
are conceptualized and implemented too often as end products rather 
than as tools to improve the learning and teaching process” (p. 207). We 
need more empirical research that attends closely to whether electronic 
portfolios actually improve teaching and learning. Why should we use 
this technology? What learning or other desirable ends does it enable us 
to achieve that we couldn’t do without it? 

Assertion 3: Research on electronic portfolios should use rigorous 
methods and communicate findings in such a way that the contribu- 
tions of methods are clear. 

Leaders in the field of educational technology have called for greater 
rigor within all approaches to research (e.g., Robyler & Knezek, 2003; 
Strudler, 2003). Studies analyzed here suggest what those rigorous meth- 
ods look like when applied to electronic portfolio research. Methodolo- 
gies varied, and although some of these studies were more rigorous and 
systematic than others, all used instruments for data collection appropriate 
for their particular research questions, multiple sources of data, triangula- 
tion of evidence, and systematic analysis. 

What are some of the methods that proved useful for studying port- 
folios? The complex manner in which teachers learn through portfolios is 
documented by think-alouds (Carney, 2001), careful analysis of portfolio 
content (Avraamidou & Zembal-Saul, 2004; Hartmann, 2003), and 
observational data from school settings (Ellsworth, 2002; Kariuki, 2001). 
The unique contributions and constraints of particular technological tools 
are carefully examined and described by Avraamidou & Zembal-Saul 
(2004), Carney (2001), and Hartmann and Calandra (2004). 

Assertion 4: Research on electronic portfolios should be sensitive 
to context. 

Several of these studies paid special attention to context — analyzing 
how social interaction and the various tools available for portfolio au- 
thoring interacted in a setting (Carney, 2001; Hartmann, 2003). Others 
used long-term ethnographic methods that described not only individual 
effects, but systemic ones as well (Ellsworth, 2003). 
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Berliner (2002) has noted that “good science” requires not only trying 
to capture valid and reliable evidence of improvement, but also attempt- 
ing to explain the complex set of conditions that led to improvement. 
Electronic portfolio research must include both the analytic and systemic 
approaches recommended by Soloman (1991) — the analytic approaches 
to test specific theory, systemic approaches to capture the richness of 
events, and actions in complex social environments. 

Strudler has cited Hall and Hoard (2001) in suggesting that an in- 
novation configuration map might be used identify major components of 
innovations and describe observable variations for each component. 

Assertion 5: Research on electronic portfolios should include more 
large-scale and long-term longitudinal studies; Mixed method and 
cross-project designs ought to be deliberately planned. 

This analysis of electronic portfolio research has drawn primarily upon 
small-scale e-portfolios case studies, done during relatively short periods 
of time. More large-scale longitudinal research is needed. PT^ leaders have 
called for the synthesis of data from PT^ projects nationwide and more 
cross-project designs for meta-analysis (Thompson, 2005). Considering 
the large number of PT^ projects with electronic portfolio programs, ef- 
forts of this type would greatly further our understanding of e-portfolios 
used for preservice teachers. 

PT^ leaders (Thompson, 2005) have also recommended following 
preservice teachers from induction through year three. E-portfolio research 
ought to follow this recommendation, seeking to determine the effect 
of preservice e-portfolios on beginning teachers’ practices, and whether 
those practices result in improved P-12 student learning. 

Final Commentary 

Analysis of electronic portfolio research can tell us something about 
portfolios and about effective methods for studying them. Yet no matter 
how carefully selected, each study has limitations and deficiencies; it is 
only by amassing a body of empirical evidence that we can document 
electronic portfolios’ technical quality, implementation effects, fairness, 
feasibility, and singular tool affordances. 

Herman and Winters (1994) asserted, “Change alone is not enough — 
the quality of change and the efficacy of the new practices must be subject 
to inquiry.” Many believe that electronic portfolios have the potential to 
capture and develop teacher knowledge in ways not possible with older 
technologies — but if teacher educators and technologists fail to critically 
evaluate uses of the device, we may find that the promise of portfolios 
has been perverted, and the device soon abandoned. Only good research 
can keep electronic portfolios from being another educational fad. Let 
us continue our inquiry. 
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